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Abstract 

The  armed  forces  of  the  U.S.  military  owe  much  of  its  competitive  edge  to  the  sophisticated 
design  skills  required  to  fashion  complex  integrated  defense  systems  from  advanced 
technologies.  This  study  identifies  the  critical  attributes  of  successful  design  team  skills  and 
the  extent  they  are  maintained  through  prototyping  efforts.  Defense  system  design  and  article 
prototyping  have  been  proposed  as  a  strategy  to  maintain  the  industrial  base  design  skills 
while  foregoing  the  expense  and  production  engineering  of  full-scale  system  development 
and  deployment.  Studies  of  the  design  process  and  historical  analyses  of  defense  design  are 
examined  to  identify  the  attributes  of  successful  design  teams  and  their  skills.  The 
engineering  goals  of  prototyping  and  full-scale  development  efforts  are  discussed.  A  contrast 
is  made  with  the  skill  development  from  full-scale  system  design,  production,  and 
deployment.  The  scope  of  the  prototyping  efforts  is  compared  to  the  scope  of  full-scale 
system  development.  Design  skills  for  prototyping  and  full-scale  efforts  are  then  compared, 
and  an  assessment  is  developed  on  the  efficacy  of  prototyping  to  maintain  necessary  design 
skills.  Implications  for  defense  acquisition  and  engineering  are  discussed. 

Introduction 

The  political  and  military  environment  after  World  War  II  laid  down  the  architecture  of 
a  U.S.  defense  establishment  that  would  endure  for  over  60  years  and  provide  a  relatively 
stable,  if  challenging,  framework  for  developing  and  acquiring  weapon  systems.  Despite  its 
challenges,  the  current  acquisition  framework  has  consistently  produced  systems  of  clear 
technical  superiority  over  almost  all  of  its  international  competitors. 

The  performance  gap  between  U.S.  systems  and  the  rest  of  the  world  is  closing 
faster  than  at  any  point  since  the  end  of  World  War  II.  Near-peer  U.S.  competitors  have 
apparently  managed  to  replicate  some  of  the  most  advanced  U.S.  capabilities  (Majumdar, 
2014).  Senior  leadership  in  the  Department  of  Defense  (DoD)  is  calling  for  a  Long  Range 
Research  and  Development  Plan  that  will  open  new  avenues  of  competitive  advantage  for 
U.S.  armed  forces.  Similar  efforts  drove  changes  in  strategic  forces  to  nuclear  weapons  in 
the  1950s  and  advanced  capabilities  such  as  precision  guided  munitions  and  stealth  in  the 
1970s  (Hagel,  2014).  Coupled  with  the  steady  implementation  of  the  Better  Buying  Power 
initiatives  (OUSD[AT&L],  2014),  the  expected  end  state  is  more  efficient  acquisition 
processes  introducing  a  new  wave  of  technology  into  weapons  acquisition  programs. 

A  prominent  element  of  this  strategy  is  the  reliance  on  prototyping  to  maintain  the 
development  competency  of  industrial  design  teams.  Such  a  strategy  hinges  on  the 
assumption  that  design  skills  can  be  preserved  through  prototype  development  while 
avoiding  the  much  larger  cost  of  engineering  development  and  production.  The  logic  is 
appealing;  however,  the  utility  of  prototyping  has  had  a  demonstrably  mixed  record  in 
defense  acquisition.  Some  programs,  such  as  the  Manhattan  Project,  were  critically 
dependent  on  the  engineering  lessons  of  their  prototype  reactors  (U.S.  Department  of 
Energy,  2010).  In  contrast,  an  analysis  of  the  F-117  program  indicates  that,  despite  its 
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remarkable  HAVE  BLUE  prototyping  effort,  the  development  timeline  was  about  the  same 
as  its  contemporaneous  F-16  sister  project  (Smith,  Shulman,  &  Leonard,  1996). 

Understanding  the  complexities  of  this  strategy  requires  an  examination  of  three 
elements:  (1)  design  as  a  cognitive  activity  in  itself  and  traditional  use  of  prototyping,  (2)  the 
record  of  defense  prototyping,  and  (3)  the  viability  of  design  teams.  This  analysis  examines 
the  relationship  of  these  areas  to  overall  systems  development  and  prototyping  and  provides 
informed  recommendations  for  a  prototyping  strategy. 

Design  Skills  and  Prototyping 

The  popular  notion  of  the  design  process  is  one  that  begins  at  a  high  level  of 
abstraction  and  ends  in  the  technical  details  and  processes  of  making  a  designed  article. 

The  design  process  is  usually  described  as  a  series  of  sequential  steps:  (1 )  problem 
clarification,  (2)  conceptualizing  (3)  embodiments  in  layouts,  and  (4)  elaboration  and 
detailing  (Eder,  1998).  Academics  and  practitioners  both  use  the  mapping  from  the  general 
to  the  specific  alike,  in  part  because  it  is  readily  understandable.  One  such  depiction  is  given 
in  Figure  1. 


Figure  1.  Development  Model  for  Defense  Acquisition  Programs 

(OUSD[AT&L],  2015) 

Defense  acquisition  practitioners  will  recognize  the  acquisition  model  from  the  DoD 
5000.02  “Operation  of  the  Defense  Acquisition  System.”  Milestones  A,  B,  and  C  correspond 
roughly  with  Eder’s  conceptualizing,  embodiments/layout,  and  detail/elaboration  steps, 
respectively.  Prototyping  activities  are  not  explicitly  called  out  in  Figure  1 — they  are  left  to 
the  discretion  of  the  program  manager. 

Program  managers  with  any  experience  know  that  the  design  process  is  rarely  a 
steady  progression  through  milestones  or  decision  gates.  Often  the  design  process  is  full  of 
blind  alleys  as  designers  work  through  technical  challenges  or  adapt  to  changes  in  funding 
or  schedule  profiles.  Less  well  understood  is  the  nature  of  the  design  process  itself.  While  it 
is  modeled  as  a  sequential  exercise  in  problem  analysis  and  solution  synthesis,  real 
designers  work  in  a  far  more  non-linear  fashion  that  is  dictated  by  experience,  cognitive 
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framework,  and  specialized  knowledge.  These  attributes  are  often  key  in  determining  the 
need,  form,  and  function  of  a  design  prototype. 

Designers  at  Work 

Design  is  taught  as  a  progression  from  the  abstract  to  the  concrete  where  a  design 
challenge  is  bounded,  requirements  identified,  the  design  problem  decomposed,  and  its 
component  parts  analyzed.  Once  the  design  challenge  is  completely  understood,  solutions 
are  generated,  evaluated  against  requirements,  and  an  overarching  solution  concept 
chosen.  The  selected  concept  is  built  out  into  greater  and  more  concrete  detail  during 
functional  and  physical  allocation  of  requirements  and  then  implemented  with  physical 
assemblies. 

NASA  portrays  this  process  as  a  cycle,  with  the  finished  product  tested,  evaluated, 
and  lessons-learned  applied  to  design  modifications  as  appropriate.  Figure  2  is  an  idealized 
diagram  of  the  design  process. 


Figure  2.  Idealized  NASA  Design  Process 

(NASA,  2008) 

Prototyping  is  an  integral  part  of  the  design  process.  Figure  2  is  not  specific  about  the  type 
of  prototype;  however,  the  context  implies  a  full-system  prototype. 

Studies  of  actual  designers  in  action  indicate  a  less  systematic  approach  to  design, 
particularly  with  ‘ill-defined’  problems  (Cross,  2004;  Guindon,  1990).  Rather  than  the  steady 
progression  from  abstract  concept  to  physical  implementation  of  Figure  2,  designers  will 
often  opportunistically  abandon  the  classic  “breadth-first-then-depth”  approach  if  a  partial 
solution  appears  to  fill  a  known  niche  in  the  solution  space.  Partial  design  solutions  are  built 


MSi 


-303- 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


out  as  far  as  needed,  after  which  experienced  designers  often  return  to  the  higher-level 
problem  decomposition — or  move  forward  with  a  physical  prototyping  effort  at  the  full-  or 
sub-system  level  (Guindon,  1990).  This  “hop-scotching”  among  the  various  design  steps  can 
occur  throughout  the  design  process. 

The  seemingly  haphazard  execution  of  the  design  process  at  first  appears  inefficient, 
but  in  ill-defined  or  novel  design  challenges,  it  can  be  more  efficient  than  a  conventional 
breadth-to-depth  approach.  In  novel  design  challenges,  where  there  is  no  clear  abstract-to- 
detail  path,  designers  bring  their  own  professional  references  and  biases  when  mapping  out 
the  solution  space.  Experienced  designers  use  partial  solutions  and  relevant  analogies  not 
only  as  building  blocks  toward  a  comprehensive  solution,  but  also  as  a  way  to  bound  the 
design  problem  and  its  analysis/decomposition  (Kalogerakas,  Luthje,  &  Herstatt,  2010; 
Spitas,  2011).  The  partial  solutions  become  anchor  points  in  the  solution  space  for  the  final, 
comprehensive  concept. 

Practicing  designers  see  design  solutions  coalesce  around  established,  workable 
partial  solutions  rather  than  uniformly  evolving  from  methodical  solution  generation.  The  role 
of  the  prototype  is  more  complex  than  its  depiction  in  Figure  2.  The  need  for  a  prototype 
becomes  less  of  a  demonstration  of  a  complete  concept  and  more  of  a  validation  of 
contemplated,  smaller  scale  partial  solutions.  As  with  known,  validated  partial  solutions,  this 
type  of  prototyping  informs  the  design  decomposition/analysis  and  maps  the  possible 
solution  space. 

The  practice  of  the  individual  designer  becomes  more  complex  as  multiple  design 
specialties  are  combined  to  produce  the  large  systems  commonly  associated  with  defense 
programs.  Teams  of  designers  produce  more  than  physical  systems.  They  also  produce  a 
wealth  of  tacit  knowledge,  experience,  and  approaches  that  are  not  only  signature  hallmarks 
of  a  design  organization,  but  often  unique  to  the  team  itself.  How  these  teams  become,  and 
remain,  viable  influences  the  specific  role  of  prototyping  in  the  design  process. 

Design  Teams 

Defense  systems  are  the  product  of  industrial  design  teams.  Since  the  end  of  World 
War  II,  the  industrial  design  teams  have  been  the  bases  on  which  are  built  the  weapons 
systems  of  the  U.S.  Armed  Forces.  The  defense  industrial  base  of  the  immediate  post-war 
period  supported  dozens  of  large  and  small  companies  producing  basic  technology  to 
sophisticated  large  systems  integration  (Watts,  2008;  Watts  &  Harrison,  2011).  Through 
almost  70  years  of  rising  and  falling  defense  budgets  since  the  end  of  World  War  II,  the 
industrial  base  has  gradually  narrowed  to  a  small  number  of  large  system  integrators,  10-20 
secondary  vendors,  and  perhaps  an  equal  number  of  tertiary  companies  whose  primary 
business  is  defense  articles,  leading  DoD  senior  leadership  to  launch  a  new  round  of 
technology  offset  to  maintain  U.S.  technical  supremacy  (Hagel,  2014). 

DoD  leadership  places  a  high  priority  on  preserving  design  team  skills  without 
knowing  exactly  the  critical  attributes  of  their  success.  This  leads  to  well-intentioned,  but 
poorly  informed,  efforts  to  preserve  production  lines  and  design  teams  that  maintain  the 
status  quo  without  effectively  building  a  path  to  future  technical  developments  that  lead  to 
the  next  generation  of  defense  systems. 

The  quality  and  effectiveness  of  a  design  team  are  functions  of  a  number  of 
attributes.  Studies  indicate  successful  teams  are:  (1)  experienced  (Atman  et  al.,  2007; 

Cross,  2001),  (2)  well-grounded  in  the  system  architecture  under  consideration  (Brusoni  & 
Prencipe,  2011;  Henderson  &  Clark,  1990),  (3)  adept  at  using  firm-specific  knowledge  to 
achieve  competitive  advantage  (Lorell,  Saunders,  &  Levaux,  1995),  (4)  active  in  designing 
systems  and  their  components,  (5)  current  on  state-of-the-practice  technology  and 
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approaches,  (6)  intimately  familiar  with  the  design  context  and  environment  (Brusoni  & 
Prencipe,  2001;  Henderson  &  Clark,  1990),  (7)  composed  of  a  “critical  mass”  of  the  design 
disciplines  necessary  to  generate  feasible  design  alternatives  (Drezner  et  al.,  1992),  and  (8) 
able  to  evaluate  alternatives  and  find  the  most  effective  design  with  the  lowest  development 
risk  (Atman  et  al.,  2007;  Cross,  2001). 

The  strategies  of  successful  design  teams  mirror  those  of  individual  designers. 
Familiar  design  challenges  invite  known  and  proven  solutions  with  implementations  suitable 
for  the  design  context.  Complex  design  problems  are  approached  with  breadth-first 
decomposition  with  designers  opportunistically  deploying  partial  design  solutions  to  aid 
further  problem  decomposition  and  build  a  comprehensive  design  solution.  The  advantage 
of  a  well-balanced  and  experienced  design  team  is  that  it  can  draw  from  firm-specific 
knowledge  to  complement  its  general  engineering  and  system  knowledge  to  develop 
competitive  solutions  (Drezner  et  al.,  1992;  Lorell  et  al.,  1995).  General  engineering 
knowledge  of  a  design  team  informs  its  ability  to  produce  a  quality  technical  product,  while 
its  system  knowledge  narrows  the  team’s  specialty  to,  for  example,  bombers  versus  tactical 
aircraft.  The  firm-specific  knowledge  represents  the  way  a  design  team  envisions  its  solution 
and  is  based  on  its  prior  experience  and  accumulated,  proprietary  architectural  knowledge. 
This  is  seen  in  the  way  certain  design  elements  frequently  recur  in  a  company’s  product 
developments. 

Firm-specific  knowledge  also  informs  a  design  team’s  strategy  with  respect  to 
prototyping.  Experience  with  developing  specific  types  of  systems  bounds  the  design  space 
through  certain  requirements,  constraints  imposed  by  the  customer,  and  physics.  The  firm- 
specific  knowledge  imposes  cognitive  constraints  and  imperatives  on  the  design  teams  on 
how  the  design  space  should  be  met  in  achieving  a  physical  implementation  of  the  design. 
The  design  team  then  deploys  its  development  strategy,  resulting  in  a  variety  of 
instantiations  of  prototype  engineering.  The  next  section  examines  the  record  of  prototyping 
and  its  effectiveness  in  producing  successful  design  outcomes. 

Defense  Prototyping 

There  are  many  different  ways  to  define  prototypes  and  prototype  engineering.  The 
following  is  a  definition  that  can  be  used  by  the  technologist  and  production  engineer  and 
strikes  an  appropriate  balance: 

A  prototype  is  a  product  (hardware  and/or  software)  that  allows  hands-on 
testing  in  a  realistic  environment.  In  scope  and  scale,  it  represents  a  concept, 
subsystem,  or  production  article  with  potential  utility.  It  is  built  to  improve  the 
quality  of  decisions,  not  merely  to  demonstrate  satisfaction  of  contract 
specifications.  (Drezner,  1992,  p.  9) 

The  lead  production  engineer  for  a  tactical  aircraft  will  have  a  different  vision  of  a 
prototype  than  the  lead  technologist  of  a  technical  demonstrator  project.  Both  professionals 
can  effectively  use  prototyping.  The  former  may  use  the  prototyping  effort  to  validate  a 
design  prior  to  production,  while  the  latter  may  use  a  prototype  to  inform  a  pending  design 
decision.  Both  uses  are  appropriate  in  the  right  context.  Can  the  production  engineer  use  an 
EMD  prototype  to  inform  design  decisions?  It  is  certainly  possible — this  is  the  acquisition 
strategy  of  the  F-35  with  its  parallel  design  and  production  efforts.  Can  the  technologist 
make  production  decisions  from  a  technology  demonstrator?  This  approach  was  used  in  the 
F-1 17  development.  When  the  critical  low-observable  characteristics  were  understood  and 
marginally  producible,  the  aircraft  was  pushed  into  production  (Smith  et  al.,  1996).  Both 
programs  used  their  respective  prototypes  to  inform  subsequent  decisions. 
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Prototyping  has  had  a  mixed  history  in  defense  acquisition  programs,  in  part 
because  the  terms  “prototype”  and  “prototyping”  have  never  been  defined  in  a  consensus 
manner  (Borowski,  2012).  Researchers  and  practitioners  alike  have  used  a  variety  of 
definitions  with  varying  scope.  Most  acquisition  professionals  have  an  intuitive  feel  for  the 
attributes  of  a  prototype  and  prototyping  efforts,  but  they  are  usually  informed  by  the  context 
in  which  a  prototype  is  executed.  Not  surprisingly,  the  value  of  prototyping  is  very  much 
dependent  on  when  and  how  a  prototyping  effort  is  initiated  and  structured,  respectively. 

Prototyping  Value  Added 

The  conventional  wisdom  regarding  prototyping  is  that  it  is  always  beneficial  and  has 
positive  influence  on  program  outcomes.  The  extant  studies  on  the  value  of  defense 
prototyping  indicate  a  variety  of  program  outcomes  for  programs  that  have  used  prototypes. 
Perhaps  the  most  surprising  observation  is  that  program  outcomes  have  only  a  weak 
relationship  with  prototyping  efforts  (Arena  et  al.,  2006;  Borowski,  2012;  Drezner,  1992; 
Drezner  &  Huang,  2009).  Cost  and  schedule  growth  of  acquisition  programs  are  as  evident 
in  programs  with  prototypes  and  prototype  engineering  as  those  without. 

The  weak  relationship  is  counter-intuitive.  The  acquisition  practitioner  would  maintain 
that  prototyping  is  almost  always  a  worthwhile  risk  reduction  technique.  Examining  individual 
programs  reveals  that  the  scope  and  timing  of  prototyping  efforts  within  a  program  greatly 
affect  the  value  added  by  the  prototyping  effort  (Drezner  &  Huang,  2009;  Tyson  et  al.,  1991). 
The  type  of  prototype,  and  its  timing  within  a  program,  affects  its  value  added  to  the  program 
outcome. 

For  the  balance  of  this  paper,  prototype  articles  and  engineering  are  classed  as 
technology  demonstrators  (Pt),  developmental  prototypes  (Pd),  prototypes  of  subsystems 
(Ps),  or  the  prototype  of  a  whole  system  (Pemd).  The  designations  Pt  and  Pd  loosely  equate  to 
X-  and  Y-plane  levels  of  design  maturity.  Subsystem  prototypes,  Ps  are  efforts  restricted  to  a 
subset  of  a  system  architecture,  for  example  aircraft  avionics  or  ground  vehicle  suspension. 
The  Pemd  is  a  near-final  system  configuration  and  can  be  considered  a  very  mature  prototype 
that  addresses  the  full  spectrum  of  performance  requirements. 

The  extant  literature  would  indicated  that  the  most  effective  prototyping  efforts  are 
those  that  are  launched  in  the  context  of  a  known  and  stable  knowledge  of  the  end-state 
configuration  (Borowski,  2012;  Drezner,  1992).  From  a  classic  design  standpoint,  the 
problem  has  been  completely  decomposed,  technical  challenges  properly  identified,  and  a 
comprehensive  concept  developed.  Subsystem  prototypes  (Ps)  contribute  to  the 
development  of  a  technical  demonstrator  prototype  (Pt)  that  focuses  on  a  limited  number  of 
critical  technical  challenges.  The  technical  resolutions  and  remaining  requirements  are 
incorporated  into  a  developmental  prototype  (Pd).  The  lessons  learned  from  Pd  are  reflected 
in  Pemd-  Table  1  summarizes  this  progression  and  integrates  the  design  cycle  of  Figure  1 
with  the  preceding  discussion. 
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Table  1.  Classic  Design  Engineering  Prototyping 


Design  Phase 

Goal 

Prototyping 

Remarks 

Identify  Problem 

Translate  problem 
statement  into 
design  space 

Parametric 
analysis;  models 
and  simulation 

Broad  analysis  of  the  problem 

Identify  constraints 
and  requirements 

Scope  the  design 
space 

Parametric 
analysis;  models 
and  simulation 

Critical  technical  areas  identified: 
prototyping  done  with  digital 
simulations  and  models 

Idea  generation 

Breadth-first 
problem  analysis 
and  decomposition 

Focused  digital 
simulations  of 
critical  subsystems 

Critical  subsystem  challenges 
identified;  initial  functional  allocations 
developed 

Evaluate  ideas  Feasibility  Subsystem  Key  subsystem  challenges  identified; 

and  select  an  evaluation  with  prototypes  -  Ps  Ps  developed  at  TRL  3-5 

approach  depth  analysis  of 


potential  solutions; 
architectural  and 
system  selection 


Model/Prototype 

Specify  and  build 
an  appropriate 
prototype  for 
testing 

Technology 
demonstrator 
prototype  -  Pt 

Ps  results  integrated  up  to  a  Pt  effort 
at  TRL  5-6;  "X-plane” 

Refine  design 

Move  design 
forward  for 
production 

Developmental 
prototype  -  Pd 

P;  results  integrated  with  more 
defined  requirements  to  a  Pd  at  TRL 
6-8;  ‘Y-plane’ 

Test/evaluate 

Test  of  the  EMD 
article 

Production 
prototype  -  Pemd 

Integration  into  a  near-production 
end-item  configuration;  LRIP 
configuration 

Table  1  outlines  the  situation  where  the  end  configuration,  and,  consequently,  the 
instantiation  of  Pemd,  is  well  defined.  The  evolution  of  the  F-16  Falcon  reflects  the 
progression  of  Table  1.  The  Pemd  configuration  changed  very  little  from  the  Pd,  “Y-plane,” 
prototype  and  resulted  in  a  relatively  trouble-free  operational  introduction  and 
maintainability.  The  maintenance  to  flight  hour  ratio  for  the  production  aircraft  was  very 
similar  to  that  of  Pemd  (Smith  et  al.,  1996). 

The  F-16  prototyping  approach  serves  well  with  a  closed  design  process  (Dc),  where 
the  Pemd  has  a  well-understood  configuration.  Advancing  prototype  engineering  where  the 
end  configuration  is  undefined,  or  open  design  (D0),  can  have  any  number  of  unintended 
consequences.  The  early  engineering  for  the  F-1 1 7  stealth  fighter  focused  almost  entirely  on 
an  aircraft  plan  form  optimized  for  low  radar  cross-section  (RCS),  even  though  the  design 
was  closed  in  the  sense  that  the  end  configuration  was  known  to  be  a  manned  penetrator 
aircraft.  Once  the  low  RCS  performance  was  demonstrated,  these  key  elements  of  the  Pt 
were  transitioned  directly  to  Pemd.  Key  design  elements  that  would  have  been  demonstrated 
in  a  Pd,  such  as  logistic  support  and  maintainability,  were  notably  absent.  As  a  result,  the 
maintenance  of  the  F-1 1 7  per  flight  hour  greatly  exceeded  the  F-1 6  and  other  contemporary 
aircraft  (Smith  et  al.,  1996). 

The  designers  of  the  first  Plutonium  production  reactors  faced  a  similar  leap  from  Pt 
to  Pemd,  but  within  an  open  design,  D0,  context.  The  end  configuration  was  almost  completely 
undefined.  The  first  production  reactors  for  the  Manhattan  Project  suffered  a  near- 
catastrophic  engineering  design  flaw  stemming  directly  from  the  leap  between  Pt  and  Pemd. 
Enrico  Fermi’s  Chicago  Pile-1  (CP-1 )  demonstrated  the  first  sustainable  nuclear  reaction  in 
late  1942 — the  primary  focus  of  the  effort;  however,  the  CP-1  did  little  to  inform  the  Pemd  for 
the  DuPont  designers  charged  with  building  a  production  reactor,  other  than  demonstrating 
a  sustainable  chain  reaction.  The  first  production  reactor,  the  Hanford  B-Reactor,  went 
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critical  only  to  shut  down  within  a  few  hours  because  of  Xenon  poisoning,  a  known  by¬ 
product  of  nuclear  fission.  Fortunately,  the  DuPont  engineers  had  built  in  extra  critical 
capacity  that  enabled  the  design  to  successfully  operate  once  the  failure  mechanism  was 
understood.  Simply  scaling  up  Pt  left  unanswered  vital  technical  questions  critical  to  the 
effective  operation  of  Pemd  (U.S.  Department  of  Energy,  2010). 

Prototypes  Are  Value-Added — In  the  Right  Context 

Examining  defense  acquisition  programs  using  the  Ps  to  Pemd  spectrum  and  the  idea 
of  Do  and  Dc  reveals  that  virtually  all  programs  engage  in  some  type  of  prototyping.  The  F-35 
program  is  ostensibly  a  closed  design,  although  its  parallel  design  and  manufacturing  model 
means  that  production  aircraft  of  the  current  tranche  are  Pemd  for  later  production  tranches 
as  design  efforts  move  forward.  The  DDG-1000  program  is  also  a  closed  design  in  that  the 
end  configuration  was  always  a  surface  combatant;  however,  it  brought  together  a  number 
of  subsystem,  Ps,  prototypes,  such  as  integrated  electric  drive,  a  new  self-defense  radar,  low 
RCS  and  infrared  characteristics,  and  an  advanced  gun  system  for  later  integration  into  a 
coherent  whole.  Programs  use  prototypes,  but  to  what  end? 

In  defense  acquisition  programs,  the  role  of  the  prototype  is  either  (1)  the  validation 
of  earlier  design  decisions,  (2)  a  mechanism  for  addressing  specific  technical  challenges 
that  are  critical  to  the  program,  (3)  the  verification  of  contract  specifications,  or  (4)  enhance 
competition  and  support  acquisition  decisions.  It  can  be  argued  that  programs  already  do  a 
fair  amount  of  prototyping  under  (1)  and  (2)  with  Ps  and  Pt  efforts;  however,  the  conventional 
way  of  thinking  about  prototyping  centers  around  the  Pd  and  Pemd  articles  that  are  at  the 
heart  of  the  “fly  before  you  buy”  approach.  This  is  not  a  new  concept,  but  it  has  recently 
been  reinforced  from  senior  DoD  leadership. 

In  2007,  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
(USD[AT&L])  mandated  the  use  of  competitive  prototyping  with  the  goal  of  lowering 
development  costs  by  using  competition  at  the  pre-Engineering  and  Manufacturing 
Development  (pre-EMD)  stage  (Young,  2007).  The  missive  is  notable  in  that  it  explicitly 
defines  the  role  of  prototyping  within  the  defense  acquisition  system.  Design  teams  were  to 
henceforth  focus  on  producing  designs  for  manufacture  and  production,  and  competitive 
prototyping  was  the  means  to  that  end.  The  instruction  goes  as  far  as  directing  that  “large 
teams  should  be  producing  detailed  manufacturing  design — not  solving  myriad  technical 
issues.” 


The  2007  letter  assigned  a  definitive  and  explicit  role  for  Pd  and  Pemd  prototyping 
efforts  within  the  defense  acquisition  systems.  The  intent  was  to  curb  the  cost  and  schedule 
growth  of  programs  that  were  spending  an  increasing  amount  of  time,  talent,  and  funding  to 
mature  technologies  instead  of  producing  designs  for  manufacture  (GAO,  1999,  2009).  The 
unintended  consequence  was  to  marginalize  the  idea  of  prototyping,  particularly  at  the  Ps 
level,  as  a  way  to  inform  design  decisions. 

The  2007  mandate  is  effective  where  there  are  a  variety  of  competitors  with  the 
ability  to  produce  Pd  and  Pemd  prototypes  as  part  of  a  competition.  Unfortunately,  this  has  not 
been  the  case  for  some  decades  (Watts,  2008;  Watts  &  Flarrison,  201 1 ).  The  number  of 
companies  capable  of  large  systems  integration  into  a  comprehensive  weapon  system  is 
few.  Even  if  there  were  a  healthy  number  of  competitors,  the  cost  of  structuring  three  or 
more  competitions  among  Pd/Pemd  articles  would  be  prohibitive  (Overstreet,  Bates,  & 
Mallicoat,  2013).  The  following  figure  illustrates  the  relative  costs  of  developing  prototypes 
along  the  Ps  through  Pemd  spectrum. 
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Figure  3.  Program  Cost  Versus  Prototyping  Phase  for  Defense  Acquisition 

Programs 

Figure  3  indicates  an  order  of  magnitude  higher  cost  for  each  phase  of  prototyping. 
Subsystem  prototypes  are  roughly  in  the  millions-of-dollars  range,  while  more  complex 
prototypes  at  the  X-  and  Y-plane  level  are  $1 0— $1 00  million-dollar  efforts.  A  Pemd  competitive 
prototype  that  would  meet  the  intent  of  the  2007  directive  would  likely  be  a  $1  billion  effort. 
The  estimates  are  low,  although  the  exact  numbers  are  secondary  to  the  overall  shape  of 
the  cost  curve. 

The  “fly  before  you  buy”  strategy  makes  sense  when  multiple  vendors  can  produce 
Pemd  articles  at  an  affordable  cost.  For  some  defense  industrial  base  sectors,  this  is 
impractical,  as  there  is  little  competition  remaining.  The  tactical  aircraft  “gene  pool”  is  now  so 
restricted  that  a  Pemd  type  of  competition  makes  little  sense  and  would  not  effectively  inform 
a  competition  (Borowski,  2012;  Drezner  et  al.,  1992;  Lorell  etal.,  1995). 

The  2007  direction  emphasizes  prototyping  as  a  path  to  quality  manufacturing  design 
and  admonishes  design  teams  to  avoid  “solving  myriad  technical  issues.”  This  stands  the 
whole  nature  of  prototyping  on  its  head.  Design  teams  are  most  certainly  in  the  business  of 
solving  the  technical  issues  to  get  to  the  point  of  being  able  to  integrate  subsystems  and 
assemblies  into  larger  designs.  The  skills  developed  in  prototyping  at  the  Psand  Pt  levels  are 
necessary  to  produce  prototypes  at  Pd  and  Pemd.  The  next  section  discusses  how 
prototyping  at  the  various  levels  can  be  used  to  preserve  design  skills. 

Preserving  Design  Skills  Through  Prototyping 

The  DoD  is  embarking  on  a  new  initiative  to  establish  a  technical  offset  strategy  that 
will  maintain  the  U.S.  technology  lead  in  defense  systems.  Similar  to  the  offset  strategies  of 
the  1950s  and  1970s,  this  one  is  meant  to  identify  critical  technology  areas  where  U.S. 
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dominance  is  strategically  important.  It  also  emphasizes  prototyping  to  maintain  the  skills  of 
industrial  design  teams  (Weisberger,  2014).  This  paper  has  so  far  discussed  how  designers 
work,  some  proposed  attributes  of  effective  design  teams,  and  prototyping  in  the  context  of 
defense  acquisition.  At  what  level  of  prototyping  do  design  teams  get  the  most  benefit  from 
the  prototyping  effort? 

Prototyping  Benefits  to  Design  Team  Attributes 

The  Ps-Pemd  and  Dc/D0  conceptual  framework  makes  it  possible  to  qualitatively 
identify  which  design  team  attributes  are  strengthened  by  various  types  of  prototyping  as 
shown  in  Table  2. 

Table  2.  Prototyping  Efforts  Contribution  to  Design  Team  Attributes 


Ps 

Pt 

Pd 

Perr>d 


The  rows  of  Table  2  are  the  types  of  prototyping  activities  and  their  qualitative 
contribution  to  the  desired  attributes  of  a  design  team  that  produces  an  operationally 
realized  design,  that  is,  one  that  is  in  service  and  deployed.  The  columns  are  the  previously 
identified  design  team  attributes.  The  yellow  cells  indicate  no  strong  benefit  to  a  design  team 
attribute,  while  the  green  cells  represent  positive  contributions  to  those  attributes. 

Not  surprisingly,  engaging  in  a  Pemd  effort  contributes  to  the  health  of  design  teams, 
as  it  exercises  or  takes  advantage  of  all  attributes.  It  not  only  exercises  the  complete  design 
cycle,  but  also  leverages  experience,  firm-specific  knowledge,  and  the  design  aspects  of 
production,  manufacturing,  and  logistics  engineering  required  in  a  deployed  system.  The  Pd 
level  is  almost  as  effective  as  Pemd;  however  without  logistics,  manufacturing,  or  production 
design  required,  the  size  of  a  Pd  team  may  be  much  less  than  for  a  Pemd  effort.  How  much 
less  is  a  matter  of  debate.  Drezner  (1992)  estimates  that  the  minimum  size  of  a  design  team 
can  be  as  low  as  75%  of  a  full  production  and  maintenance  team,  although  this  was 
estimated  in  the  context  of  a  large  and  competitive  tactical  aircraft  industrial  base. 

At  the  Pt  and  Ps  levels,  the  positive  effects  on  design  teams  are  less  because  the 
scope  of  the  effort  is  lower.  The  Pt  effort  is  focused  on  specific  technical  challenges  where 
the  architecture  of  the  end  system  and  the  efficiency  of  the  design  solutions  are  secondary. 
The  team  size  of  producing  a  Pt  X-plane  article  is  likely  less  than  producing  a  Pd  Y-plane. 
Experience,  leveraging  firm-specific  knowledge,  and  technical  currency  are  positively 
exercised.  Developing  Ps  articles  at  the  subsystem  level  likely  does  not  contribute  to  the 
attributes  of  a  product  design  team,  but  does  contribute  to  the  number  of  partial  solutions 
available  to  a  design  team. 

Combining  the  assessments  of  Table  2  with  the  cost  curve  of  Figure  3  confirms  what 
is  intuitive  to  most  experienced  acquisition  practitioners — producing  increasingly 
operationally  representative  prototypes  is  only  achievable  through  increased  costs.  The  new 
insight  is  that  the  various  attributes  of  an  effective  design  team  are  increasingly  exercised  at 
the  higher  levels  of  system  integration  complexity  seen  in  operationally  representative 
prototypes.  The  more  complete  exercise  of  the  design  cycle  involves  multiple  types  of 
design  expertise,  from  the  conceptual  to  the  logistic  and  operational. 

Extrapolating  the  trend  would  indicate  that  Pemd  prototyping  is  the  only  way  to 
maintain  design  team  viability.  This  may  be  true  in  narrow  cases,  but  it  ignores  the 
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conceptual  divide  between  Ps/Pt  and  Pd/Pemd  efforts.  At  the  lower  levels  of  prototyping,  the 
end  design  configuration  is  secondary  to  the  specific  technical  challenges  addressed.  Final 
design  architecture  is  not  known — it  is  a  D0  environment.  The  design  team  attributes  of  size, 
system  architecture  knowledge,  design  team  activity,  and  solution  efficiency  are  less 
important  than  experience,  firm-specific  knowledge,  and  technology  currency.  At  higher 
levels,  in  a  Dc  environment,  the  architectural  knowledge  of  the  operational  configuration 
becomes  a  primary  consideration. 

The  effectiveness  of  Pd/Pemd  prototypes  is  the  most  in  the  Dc  environment  of  an 
acquisition  program,  an  idea  supported  by  much  of  the  extant  literature  (Arena  et  al.,  2006; 
Borowski,  2012;  Coble  et  al.,  2014;  Drezner,  1992;  Drezner  et  al.,  1992;  Drezner  &  Huang, 
2009).  It  is  also  the  environment  addressed  by  the  OUSD(AT&L)  2007  directive  on 
competitive  prototyping  (Young,  2007).  The  Ps/Pt  efforts  are  most  effective  in  D0 
environments  where  the  novelty  of  solutions  and  focus  on  technical  challenges  are  critical. 
Historically,  the  D0  context  has  been  the  environment  of  the  technologist  and  basic  science 
researcher,  although  many  programs  have  found  it  necessary  to  mature  key  technologies  in 
addition  to  developing  product  architectures  and  designs  (GAO,  2006,  2009;  Studt,  2006). 
Three  approaches  to  prototyping  and  its  design  team  benefits  are  discussed  in  the  following 
paragraphs. 

The  Design  Bureau  Approach 

The  simplest  way  to  maintain  an  existing  design  team  is  to  keep  it  working  on 
developing  new,  or  improving  old,  designs.  Congressional  funding  has  often  been  used  to 
maintain  existing  production  lines  and  design  teams,  even  when  the  end  article  is 
superfluous.  The  United  States  already  realizes  this  situation  with  the  M1A1  Abrams  tank. 
New  variants  are  being  ordered  to  maintain  the  single  production  plant  in  Lima,  Ohio,  even 
though  the  Army  has  sufficient  quantities  to  meet  its  needs  for  the  foreseeable  future.  In 
many  respects  it  is  analogous  to  the  old  Soviet  approach  of  ordering  another  tactical  fighter 
or  bomber  from  the  Mikoyan  or  Tupelov  design  bureau. 

Fully  exercising  a  design  team  in  this  way  would  require  design  scope  and  effort 
resulting  in  a  Pemd  prototype  and  its  attendant  costs.  The  positive  attributes  would  be  that  the 
design  team  would  be  working  on  a  closed  design  with  a  known  architecture  with  perhaps 
minimal  improvement.  The  negative  aspects  are  almost  the  mirror  image  of  the  positive. 
Technical  innovation  is  hampered  as  designers  work  with  their  known  and  familiar  technical 
solutions  in  maintaining  a  closed  design.  Without  adjudication  of  new  requirements  or 
system  architecture,  the  end  configuration  is  likely  to  be  very  similar  to  the  legacy  article. 

Technical  Demonstrators — A  Few  Challenging  Problems 

An  alternate  strategy  is  to  move  down  the  prototyping  spectrum  and  focus  on  test 
articles  that  address  critical  technology  needs.  The  implicit  assumption  is  that  designers 
would  be  working  in  a  D0  environment,  where  a  final  system  configuration  is  left 
unaddressed.  Design  teams  in  this  type  of  prototyping  are  free  to  utilize  innovative  partial 
solutions  that  address  the  main  technical  challenge  without  the  need  to  compromise  with 
production  constraints.  In  a  D0  environment,  where  there  is  no  dominant  legacy  system 
architecture,  multiple  design  teams  are  free  to  pursue  innovative  and  potentially  disruptive 
solutions  (Christensen,  1997;  Clark,  1985;  Henderson  &  Clark,  1990).  The  cost  of 
developing  a  technology  demonstrator  prototype  can  be  an  order  of  magnitude  less  than 
that  of  a  Pd  or  Pemd  variant. 

Prototyping  can  be  cheap,  fast,  and  creative  in  the  permissive  D0  environment.  The 
main  drawback  is  that  the  technical  solutions  produced  may  have  little  alignment  with 
defense  needs  and  requirements.  Prototypes  at  the  Ps  level  produce  optimized  designs  for 
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the  subsystem  technical  challenges.  They  represent  partial  solutions  that  are  the  building 
blocks  for  system  designers,  but  are  rarely  comprehensive  to  the  point  of  underpinning 
entire  system  architectures.  Integrating  up  to  the  Pt  level  makes  for  a  more  comprehensive 
partial  solution,  but  one  that  may  still  fall  short  of  supporting  a  full  system  architecture,  as 
was  seen  in  the  F-1 1 7  and  Manhattan  Project  development  efforts. 

Architectural  Prototyping 

Simply  maintaining  design  teams  or  developing  unfocused  prototypes  is  not  the 
answer.  The  former  puts  existing  architectures  and  thinking  in  stasis  and  expends  resources 
against  design  teams  that  become  increasingly  irrelevant  as  their  architectural  knowledge 
becomes  dated  and  stale.  The  latter  is  essentially  the  technology  transition  problem  the  DoD 
now  faces  with  commercial  technologies.  They  represent  advanced  partial  solutions  to 
defense  design  problems;  however,  in  a  D0  environment  there  is  no  unifying  architectural 
concept  to  bridge  the  “valley  of  death”  between  laboratory  bench  and  acquisition  program 
(Beard  et  al.,  2009). 

The  presence  or  absence  of  system  architecture  determines  whether  the  prototyping 
activity  is  in  the  science/technology  domain  (a  D0  environment)  or  the  acquisition  domain  (a 
Dc  environment).  The  presence  of  an  architectural  framework  facilitates  the  development  of 
Pd  and  Pemd  prototypes  as  it  guides  a  design  team’s  approach  to  problem  decomposition, 
depth  of  analysis,  and  choice  of  partial  solutions  used  to  arrive  at  a  complete  design.  The 
system  architecture,  like  the  architecture  of  a  building,  possesses  design  elements  and 
motifs  that  can  be  judiciously  extended  and  modified  to  extend  the  utility  of  the  architecture. 
With  dated  architectures  there  is  little  flexibility  to  pace  the  threat  and  operational 
environment;  the  design  team  is  challenged  to  improve  the  design  without  breaking  the 
system  architecture. 

Design  teams  can  remain  technically  current  and  active  by  considering  the  design 
and  prototyping  of  the  architectures  themselves.  Developing  and  prototyping  system 
architecture,  rather  than  a  specific  design,  offers  the  design  team  the  opportunity  to 
integrate  state-of-the-practice  technology  in  new  ways  that  meets  overall  capability 
requirements.  The  quality  metric  of  a  prototype  architecture  would  be  its  ability  to  deliver 
capability  as  its  base  technologies  change  over  time.  A  suitably  flexible  architecture  would 
yield  a  number  of  specific  designs  depending  on  the  design  team  selection  of  specific  Ps 
partial  solutions  chosen  by  the  design  team. 

This  architectural  prototyping  is  similar  to  Pt  prototyping;  however,  it  works  on  a 
higher  level  than  that  of  a  product  design.  For  example,  the  F-1 17  prototype  integrated 
existing  technology  in  support  of  a  low-observable  design.  An  architectural  prototype  would 
develop  a  low-observable  architecture  broader  than  the  design  of  a  manned  tactical  fighter. 
Designers  would  consider  component  technologies  and  their  associated  technology 
trajectories  to  avoid  point  solutions  with  limited  futures.  Different  plan  forms  and 
technologies  would  be  compared  as  part  of  a  low-observable  architecture  that  offered 
varying  levels  of  stealth  capability. 

Harmonizing  Prototyping  Approaches 

The  prototyping  approaches  discussed  exercise  design  skills  but  at  different  levels. 

Ps  and  Pt  efforts  operate  at  the  low  to  middle  region  of  the  design  scale,  while  Pd  and  Pemd 
efforts  work  most  effectively  at  the  more  comprehensive  end.  Prototyping  at  the  subsystem 
and  tech  demo  level  are  the  basic  elements  of  more  complex  systems,  relatively  cheap  to 
achieve,  but  unfocused  without  an  overarching  architecture.  The  more  complex  prototypes 
are  orders  of  magnitude  more  expensive. 
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The  most  ephemeral  skill  maintained  by  higher  order  prototypes  is  the  integrative 
facilities  of  the  design  team  and  its  ability  put  together  complex  systems  into  a  unified  whole. 
Prototyping  architectures,  rather  than  an  individual  design,  can  provide  the  design  team  with 
developing  a  capability  architecture  that  can  accommodate  a  number  of  specific  designs 
that  meet  the  same  capability  requirement. 

Prototyping  architectures  place  more  emphasis  on  general-  and  system-specific, 
rather  than  firm-specific,  knowledge.  This  would  seem  to  put  industrial  design  teams  at  a 
disadvantage,  but  there  is  no  reason  to  believe  industry,  with  the  proper  incentive,  could  not 
engage  in  more  abstract  design  activities.  Designing  at  the  architecture  level  removes  some 
of  the  intellectual  property  issues  that  have  become  increasingly  common  with  government 
and  industry  cooperation.  Expanding  the  number  of  potential  architectures  multiplies  the 
number  of  potential  designs  and  design  solutions,  exercises  many  of  the  design  skills  noted 
in  this  paper,  and  makes  for  a  richer  design  environment  at  the  beginning  of  acquisition 
programs. 
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